State-space models have gained popularity in sequence modelling due to their simple and efficient network structures. However, the absence of nonlinear activation along the temporal direction limits the model's capacity.

These authors contributed equally and are ordered alphabetically.

These authors contributed equally and are ordered alphabetically.

Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads.

We thank the reviewers for their time and helpful feedback. Below we respond to their comments in turn. 'permutation invariance in write values' Y es, we process the values in parallel and then take the sum over the batch We will make this clearer in the updated manuscript. We agree this discussion would be useful. 'Error bars for the plots in Figure 2.' We will conduct multiple runs and include error bars in Figure 2. 'Add titles to each subgraph in Figure 5.' Indeed, we should have done this.

We leverage training and algorithmic techniques from metalearning to update the neural memory function in one shot.

The evolution of sequence modeling architectures, from recurrent neural networks and convolutional models to Transformers and structured state-space models, reflects ongoing efforts to address the diverse temporal dependencies inherent in sequential data. Despite this progress, systematically characterizing the strengths and limitations of these architectures remains a fundamental challenge. In this work, we propose a synthetic benchmarking framework to evaluate how effectively different sequence models capture distinct temporal structures. The core of this approach is to generate synthetic targets, each characterized by a memory function and a parameter that determines the strength of temporal dependence. This setup allows us to produce a continuum of tasks that vary in temporal complexity, enabling fine-grained analysis of model behavior concerning specific memory properties. We focus on four representative memory functions, each corresponding to a distinct class of temporal structures. Experiments on several sequence modeling architectures confirm existing theoretical insights and reveal new findings. These results demonstrate the effectiveness of the proposed method in advancing theoretical understanding and highlight the importance of using controllable targets with clearly defined structures for evaluating sequence modeling architectures.